Search for: All records

We introduce two complementary techniques for efficient optimization that reduce memory requirements while accelerating training of large-scale neural networks. The first technique, Subset-Norm step size, generalizes AdaGrad-Norm and AdaGrad(-Coordinate) through step-size sharing. Subset-Norm (SN) reduces AdaGrad’s memory footprint from O(d) to O(sqrt(d)), where d is the model size. For non-convex smooth objectives under coordinate-wise sub-gaussian noise, we show a noise-adapted high-probability convergence guarantee with improved dimensional dependence of SN over existing methods. Our second technique, Subspace-Momentum, reduces the momentum state’s memory footprint by restricting momentum to a low-dimensional subspace while performing SGD in the orthogonal complement. We prove a high-probability convergence result for Subspace-Momentum under standard assumptions. Empirical evaluation on pre-training and fine-tuning LLMs demonstrates the effectiveness of our methods. For instance, combining Subset-Norm with Subspace-Momentum achieves Adam’s validation perplexity for LLaMA 1B in approximately half the training tokens (6.8B vs 13.1B) while reducing Adam’s optimizer-states memory footprint by more than 80% with minimal additional hyperparameter tuning. 

Abstract Reaction of Tl(OTf) with 2 equiv of bis(diisopropylamino)cyclopropenylidene (BAC) in THF results in formation of [Tl(BAC)₂(OTf)] (1) in moderate yields. Subsequent reaction of1with [K][H₂‐9‐BBN] ([H₂‐9‐BBN]⁻ = dihydrido 9‐boratabicyclo[3.3.1]nonane) in THF results in formation of [Tl(BAC)(μ‐H₂‐9‐BBN)]₂(3), also in moderate yield. Complex3is the first reported thallium borohydride. We attribute its thermal stability to the strong donor ability of the BAC co‐ligand. Both1and3exhibit trigonal pyramidal geometries about Tl⁺in the solid‐state, indicative of the presence of stereochemically active lone pairs. The hydride environment in3is calculated to exhibit a 3.9 ppm downfield shift attributed to spin‐orbit effects from the adjacent Tl center. 

Currently, it is challenging to investigate aneurismal hemodynamics based on current in vivo data such as Magnetic Resonance Imaging or Computed Tomography due to the limitations in both spatial and temporal resolutions. In this work, we investigate the use of modal analysis at various resolutions to examine its usefulness for analyzing blood flows in brain aneurysms. Two variants of Dynamic Mode Decomposition (DMD): (i) Hankel-DMD; and (ii) Optimized-DMD, are used to extract the time-dependent dynamics of blood flows during one cardiac cycle. First, high-resolution hemodynamic data in patient-specific aneurysms are obtained using Computational Fluid Dynamics. Second, the dynamics modes, along with their spatial amplitudes and temporal magnitudes are calculated using the DMD analysis. Third, an examination of DMD analyses using a range of spatial and temporal resolutions of hemodynamic data to validate the applicability of DMD for low-resolution data, similar to ones in clinical practices. Our results show that DMD is able to characterize the inflow jet dynamics by separating large-scale structures and flow instabilities even at low spatial and temporal resolutions. Its robustness in quantifying the flow dynamics using the energy spectrum is demonstrated across different resolutions in all aneurysms in our study population. Our work indicates that DMD can be used for analyzing blood flow patterns of brain aneurysms and is a promising tool to be explored in in vivo.